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Chapter I 



Background 



A. Policy Problem and Research Question 

Public school teachers are usually paid according to two objective criteria: years of 
experience and educational attainment (certificates, degrees, or coursework). This system, 
known as the uniform salary schedule, has received criticism for being unfair because it does 
not reward effort or skill, and for being inefficient because it does not encourage hard work 
or attract talent (Hanushek 1981). 

Education policymakers seeking to reform the way teachers are paid have tried many 
times, often without success, to tie teacher compensation more closely to the quantity and 
quality of teachers’ work. An influential 1983 report by the National Commission on 
Excellence in Education, A Nation at Risk, shined a spotlight on the problem and spurred a 
wave of reforms during the mid- to late 1980s. Many of these included career ladders for 
teachers, which allowed them to advance in salary based on factors other than seniority such 
as demonstrated skills or performance. Elowever, most of the reforms enacted in the late 
1980s did not last very long (Glazerman 2004). This study focuses on one important 
exception, a teacher career ladder program that the state of Missouri started in 1986 and 
continues to operate more or less unchanged to this day. 

One goal of Missouri’s Career Ladder program (CL) is to help school districts that have 
difficulty retaining teachers, particularly those that are small and rural, by offering their 
teachers opportunities to earn extra pay for extra work and professional development. 
Eligibility for these opportunities is based on a combination of seniority and subjective 
performance evaluation. The policymakers who established the program hoped that the 
incentives created by these opportunities would make teaching in their district more 
attractive and consequently boost recmitment and retention. 

This paper seeks to estimate the effect that CL has had on teachers’ career decisions, 
specifically their decisions to stay in a specific school district or to remain in the teaching 
field. 
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To date, there has not been much evidence on the effectiveness of teacher incentive 
programs in general. Reviews by Glazerman (2004), Glazerman et al. (2006), Goldhaber and 
Anthony (2007), and Podgursky and Springer (2007) indicate that attempts to study teacher 
incentive programs rigorously are frequently thwarted by the early termination of the very 
programs being studied. Arizona is the only state besides Missouri to have a career ladder 
program that has lasted since the 1980s. Dowling et al. (2007) studied the effects of 
Arizona’s Career Ladder Program on student achievement. Their study design compared 
student performance in participating districts with performance in a matched set of 
comparison districts over a two-year period. They found positive impacts on test scores in 
math, reading, and writing. It is worth noting, however, that unlike the Missouri program, 
the Arizona career ladder allows student achievement to be considered in determining 
teacher pay. 

B. Overview of the Missouri Career Ladder Program 

As background, we describe the program as it operates, based on available program 
documents and published literature, and to a lesser extent how it operates in practice. More 
detailed information on program rules and operations can be found in Silman and 
Glazerman (2009). 

Program Structure and Operation. Through the CL program, teachers who meet 
statewide and district-level performance criteria are eligible to receive supplementary pay for 
meeting CL responsibilities, which can be extra teaching work or participation in 
professional development. The program does not replace the regular salary schedule. CL 
responsibilities must be academic in nature and directly related to the improvement of 
programs and services for students. 

A teacher moves up the career ladder in three stages. Teachers are assessed at each stage 
through periodic observations and evaluations of documentation. Each successive stage 
offers the opportunity to receive more supplementary pay for CL responsibilities: up to 
$1,500 for Stage I, $3,000 for Stage II, and $5,000 for Stage III. Out of more than 65,000 
teachers in 524 districts in Missouri, over 17,000 teachers (26 percent) from 333 districts (64 
percent) participated in the CL program during the 2005-06 school year. 

Teacher Eligibility and Qualifications for a Bonus. To enroll in the CL and qualify 
for bonuses, each teacher must develop a Career Development Plan (associating each CL 
responsibility with either a designated plan or some other instructional improvement). A 
district committee must then approve the teacher’s plan. Through scheduled and 
unscheduled observations, as well as reviews of their career development plan and other 
documentation such as lesson plans, the teacher must show evidence of performance at or 
above the expected level on 20 criteria listed in the district’s Performance-Based Teacher 
Evaluation (PBTE) instrument. The criteria span six areas: (1) engaging students in class, (2) 
assessing students correctly, (3) exhibiting content knowledge, (4) showing professionalism 
in the school, (5) participating in professional development, and (6) adhering to the district’s 
education mission. There are also specific qualification criteria for each stage of CL, as 
shown in Table 1. 
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Table 1. Qualifications for Missouri Career Ladder by Stage 



Qualification 


Stage 1 


Stage II 


Stage III 


Minimum experience to qualify (years) 


5 


7 


10 


Percent of PBTE criteria at “expected” level 


100 


100 


100 


Percent of PBTE criteria “above expected” level 


0 


10 


15 


Minimum number of responsibilities 


2 


3 


4 


Minimum hours 


60 


90 


120 


Maximum bonus 


$1,500 


$3,000 


$5,000 


Implied maximum hourly rate 


$25 


$33 


$42 



Note: PBTE = Performance Based Teacher Evaluation 



To receive a salary supplement, teachers must spend a specified amount of time on a 
certain number of responsibilities or professional development outside of their contracted 
time (amounts depend on the stage). Examples of the extra responsibilities that CL teachers 
undertake include providing students with opportunities for enhanced learning experiences, 
remedial assistance, and various extended day/year activities. Professional development 
could include taking college classes, attending workshops, and participating in professional 
organizations. 1 

In the 2005-06 school year, an average of 79 hours were spent by Stage I participants, 
111 hours by Stage II teachers, and 144 hours by Stage III teachers. These hours translate to 
supplementary pay of about $19, $27, and $35 per hour, respectively, for Stages I, II, and III, 
somewhat lower than the nominal hourly rates that would be earned by doing the minimum 
requirement: $25, $33, and $42 per hour. The bonus amounts have never been increased or 
adjusted for inflation since the program was established in 1985. 

District Participation. Missouri’s program is available statewide but individual school 
districts must decide if they will participate and, if so, provide matching funds. The matching 
rate varies based on district characteristics. Because the matching rate is so critical to 
understanding why a district participates, it is important to understand how they are 
determined. 

The matching rate formula in effect from the program’s inception through the 1 995 — 
1 996 school year was based entirely on the district’s assessed property value per pupil. 
Districts were ranked on this measure from lowest to highest and divided into 12 groups. 
Districts with the lowest per pupil value received 90 percent state funding for program 



1 The Missouri Department of Elementary and Secondary Education (DESE) recommends that teachers 
should not spend more than one-third of CL hours on college classes and workshops. 
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expenses. For each successive group of districts, the state funding share declined by 5 
percentage points, with the highest group receiving 35 percent state funding. 

In 1996 the formula changed, making the distribution narrower with fewer distinct 
matching rates; it also added total enrollment as a ranking factor. This current formula 
involves an initial sorting of districts by assessed property values per pupil, with the top 25 
percent (which we call “high wealth”) receiving a 40 percent match. Among the low wealth 
districts, the largest one-third of districts in terms of student enrollment are eligible for a 50 
percent match, and the bottom two-thirds (low wealth, low enrollment) are eligible for a 60 
percent match. 

District participation in the CL program has grown steadily since it started, although it 
grew most rapidly in the early years of the program, Table 2 shows the history of 
participation by district and teachers and the cost to the state. During the first six years, the 
program grew from 63 districts to 204 by 1992-93. After 1995-96 growth slowed, with a total 
of 333 districts participating in 2005-06 — an increase of 47 districts over a 10-year period. 
Similar patterns hold for growth over time in the number of teachers participating and in 
total state payments made for the program. 

In general, once a district begins participating in CL, it rarely withdraws from the 
program. Over the first 21 years of the program, 361 districts have participated and 23 (six 
percent) have withdrawn. 

The remainder of this paper discusses the methods, data, and findings of an analysis 
used to estimate impacts of CL on teacher mobility. The paper concludes with a summary 
and discussion of findings. 
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Table 2. Participation in and Cost of the Missouri Career Ladder Program 



Year 


Number of Districts 
Participating 


Number of Teachers 
Participating 


Total State 
Payment 


1986-87 


63 


2,400 


$2,624,025 


1987-88 


121 


5,074 


$7,182,975 


1988-89 


147 


5,811 


$10,484,500 


1989-90 


177 


6,803 


$13,839,075 


1990-91 


192 


7,580 


$16,688,675 


1991-92 


199 


8,322 


$18,902,575 


1992-93 


204 


8,536 


$20,362,750 


1993-94 


229 


10,696 


$24,426,950 


1994-95 


269 


13,021 


$29,300,325 


1995-96 


286 


14,107 


$33,358,250 


1996-97 


278 


13,741 


$34,312,899 


1997-98 


288 


14,098 


$35,799,849 


1998-99 


299 


14,707 


$37,333,522 


1999-00 


309 


15,827 


$37,687,074 


2000-01 


322 


16,688 


$37,993,100 


2001-02 


330 


17,101 


$38,253,625 


2002-03 


338 


17,412 


$38,599,500 


2003-04 


332 


16,982 


$37,103,360 


2004-05 


328 


16,919 


$36,465,400 


2005-06 


333 


17,378 


$36,986,803 



Source: This table contains data from the Missouri Career Ladder program’s 2005-06 annual 
report, produced by the Missouri Department of Elementary and Secondary Education. 
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Methods and Data 



A. Analytic Goal and Challenges 

The goal of the analysis in this paper is to estimate the relationship between eligibility 
for the CL program and teachers’ career decisions. If CL induces teachers to remain in their 
districts and thus stay in the profession, it may reduce costs to districts (by reducing 
replacement costs) and minimize the potential disruption to students and schools caused by 
teacher turnover. 

An extensive longitudinal database on teachers throughout the state of Missouri 
provides a unique opportunity to estimate this relationship between CL and teacher mobility, 
but there are several complicating factors that must be carefully accounted for in the analysis 
and interpretation of the analytic results. 

First, not all mobility is the same. We do not have the data to measure whether mobility 
is raising or lowering the overall quality of the teaching force (through attrition of weaker 
teachers or stronger teachers), but we can measure whether the CL affects teachers at 
different stages of their career. Furthermore, some movement of teachers out of the 
classroom may be good if those individuals have been promoted to productive positions in 
administration such as principalships. Mobility findings should therefore be interpreted with 
this in mind. 

Second, there are many factors that affect teachers’ career decisions, of which teachers’ 
eligibility for CL payments is just one. If compensation and working conditions in general 
are the primary determinants, then CL eligibility is one small component of this package. 
Compensation and labor market conditions can vary across districts and within districts and 
they can vary over time, so identifying the unique effect of the CL program amongst the 
many other changes observed in the data is a challenge. This paper aims to isolate the CL 
effect by using a large dataset covering several years, and employing statistical controls as 
much as possible. 

Third, the effects of CL eligibility may not be uniform across all teachers. While we 
focus on the effect of being eligible for the program, the effects of the program may be 




7 



different at different stages of a teacher’s career. For novice teachers, the program’s 
incentives are of value for their future benefits. For mid-career teachers, the possibility of 
receiving more pay is more immediate. For veteran teachers with more than 20 years of 
experience, the effect of behavior on retirement benefits may overshadow any role played by 
pay supplements like CL that do not count toward calculation of the pension benefit 
amount. Therefore, we examined subgroups defined by teacher experience and estimated 
models with the interaction of experience and CL status. 

Finally, there is a risk that the observed relationship between CL and teacher mobility 
may be spurious, reflecting the types of districts that choose to participate in the program 
and the factors associated with the timing of that choice more than the true impact of the 
program. For example, if a participating district is hard to staff, then simple comparisons of 
retention rates between teachers in CL districts and observationally similar teachers in non- 
CL districts could understate the true impact of the program. Methods are needed to allow 
us to isolate the effect of the program from unobserved determinants of teacher mobility 
and that also explain district participation. This analysis attempts to address this through 
statistical controls (for measured variables) and instrumental variables (for unmeasured 
factors). 

B. Model and Estimation 

To examine the relationship between district CL participation and teacher mobility 
while addressing the issues discussed above, we used a flexible statistical model that is well 
suited to studying the duration of employment spells and has been used routinely to model 
teacher behavior (Stinebrickner 1998; Podgursky et al. 2004). This model, the Cox 
proportional hazard regression, has the following basic form: 

hit) = h 0 it)e b,x,+ - +bkXk 

where h(t) is the probability of exit (defined alternatively as leaving teaching or leaving 
the school district) for those reach year f, and h 0 (t) is the baseline hazard function (defined 
below). X 1 through X k are the control variables, including the district’s CL status, and b 1 
through b k are used to compute the estimated hazard ratios, measuring the change exit 
probability associated with a one-unit change in the corresponding variable X 1 through X k . 
A hazard ratio of greater than one indicates that greater values of that variable are associated 
with a higher exit probability. We estimate this model for a dataset that includes one 
observation per teacher-district spell. 

The baseline hazard function represents the percentage of teachers who experience a 
transition in year / given that they are still employed (in the district) in year (/-/). These can 
be thought of as year-by-year exit rates. The Cox model assumes that the time path of these 
exit rates is some function that shifts up or down based on the factors such as CL status, 
which we hypothesize make a difference. 

We first define the outcome to study. In doing so, we differentiate among four types of 
teacher transitions, as follows: 
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• Transitory moves. Teachers often leave teaching for one or two years and then 
return to teaching in the same district. This type of “churning” mobility is 
common and may reflect sabbaticals to raise a child. We treat transitory movers 
as if they were stayers. 

• Promotions. Teachers often leave teaching to take a different job within the 
same district. Job categories included in this group were central office 
administrator, principal or assistant principal, librarian, or guidance counselor. 
We treat promotions as censored data, meaning that the exit status 
(stayer/ mover/leaver) is unknown as of the time of the promotion. 

• Changing districts. We categorize teachers who move to a full-time position in 
a different Missouri school district as “movers.” 

• Leaving the data. Teachers no longer observed in Missouri employee data (for 
reasons other than censoring due to the final year of our data) we categorize as 
“leavers.” It is likely that these teachers have either retired, left the profession, 
taken a job at a private school, or moved out of the state. 

The empirical analysis focuses on the last two categories, movers and leavers, and the 
relationship between district CL participation and the decision to either move to a different 
district or leave teaching in the state of Missouri. 

There are several explanatory variables we would want to include in this model, but 
specifying the CL status variable is an especially important step. The main parameter of 
interest in this study is the coefficient on this variable and the corresponding hazard ratio, 
which ratio represents the relative probability of a transition for two groups (e.g., CL and 
non-CL). A hazard ratio of 1.0 means that teachers in the two groups are equally likely to 
leave (their district or the state public education system) at any given time. A hazard ratio of 
less than 1.0 means that CL-eligible teachers are less likely to leave teaching and a ratio of 
greater than 1.0 means the opposite. For instance, a hazard ratio of 0.50 on the district CL 
participation indicator would imply that teachers in participating districts are half as likely as 
teachers in non-participating districts to exit, holding all else equal. 

Our main approach was to construct the district CL participation variable as an 
indicator for the district participating in the CL program within five years following the start 
of a given teacher’s spell in the district (the spell is the time period during which that teacher 
taught in that district). - However, another plausible model would suggest that the CL 
indicator should only specify whether a district participated within a shorter or longer time 
frame (2 or 10 years, for example) of the start of the teacher’s spell. The appropriate time 



2 We define district CL participation from the beginning of the teacher’s spell in that district in order to 
treat teachers that move equivalently to teachers that stay. On average, teachers that stay have longer district 
spells. 
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horizon depends on how well we believed teachers anticipated future district policies in 
formulating their decisions. 

We also included district characteristics in the model that are likely to be associated with 
differences in teacher mobility. Time-varying control variables include the log of district 
enrollment, district percent of African-American and Hispanic students, district percent of 
economically disadvantaged students (as measured by eligibility for free or reduced price 
lunch), log of district average full-time teacher salary (in constant 2007 dollars), and cohort 
indicators based on the year the teacher started teaching. 

Time -invariant control variables include the district percentage of households with no 
college education, district percentage of households that are poor, log of median district 
household income, an indicator for the district being in a large or mid-sized city, and an 
indicator for the district being in a large or small town. 

The critical step for addressing selection bias is the use of instrumental variables. We 
included a first stage regression that predicts the probability of a district participating in the 
program using variables that are believed to be unrelated to teacher mobility. We used the 
CL matching rate based on the 1986 property value per pupil as indicators for the district 
being in the top 25 percent property value per pupil, the top 33 percent for enrollment in the 
state, and the interaction of the top wealth and enrollment categories. These variables are 
tied closely to the arbitrary jumps in matching rates that we believe influenced the likelihood 
of participating, but are not highly correlated with the mobility outcomes of interest. 

In addition to the benchmark model, we also performed several sensitivity checks to 
ensure that the results are robust. We re-estimated the model using a Weibull hazard 
function instead of the Cox model, to see how sensitive the results are to the model 
specification. 

We also estimated the model omitting teachers in non-participating districts with 
observable characteristics that were markedly different from any participating districts. We 
estimated the model omitting teachers in 38 non-CL districts with propensity scores for 
district participation that were lower than those of any participating district. 1 

Another check was to estimate the model only including the first observed teacher- 
district spell for each teacher, to test whether over-representation of highly mobile teachers 
is affecting the results. 

Finally, we estimated the model excluding the two largest school districts in Missouri, 
Kansas City and St. Louis. Unlike the rest of the state, these two districts have retirement 
systems separate from the state system, creating some unusual inter-district mobility 
disincentives (Costrell and Podgursky 2009) that threaten to mask CL effects. 



3 See Booker and Glazerman (2008) for details on the estimation of the propensity score measure. 
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Because teacher CL eligibility varies with years of experience (at least five years of 
experience to be eligible for Stage I, and at least 10 years for Stage III), mobility effects of 
district participation are likely to differ by how long the teacher has been in the district. We 
estimate separate effects for different ranges of teacher experience, defining the ranges as 
one- 5 years of experience in the district, 6-10 years, 11-20 years, and 21+ years. We split the 
teachers eligible for Stage III awards into two groups in order to differentiate teacher 
retirement effects (likely most pronounced in the 21+ years group) from other teacher 
mobility effects. 

All estimates account for the clustering of teachers within districts. Ignoring this would 
produce biased standard errors and invalid hypothesis tests. Therefore, we used the 
sandwich estimator for clustering at the district level. 

C. Data Description 

The data for this paper come from the Missouri Department of Elementary and 
Secondary Education (DESE) and includes teacher-level information on all Missouri public 
school teachers and other public school employees for the 18 school years 1989-90 through 
2006-07. DESE also provided district-level data on student demographics, finances, and 
employment. 

The teacher-level dataset contains one observation for each employee- school-year 
combination, including the district number and a unique teacher identification number, so 
we could follow teachers throughout the state if they switched districts during the school 
year and from year to year. Each employee has a job status code, which allowed us to 
identify teachers who switch to another job within the public school system, such as 
principal or librarian. Each observation includes a measure of the number of full-time 
equivalent units the teacher worked that year at that school, allowing us to identify and 
restrict our analysis to full-time teachers. In addition, each teacher observation includes their 
years of experience, both in Missouri and in a specific district (which eliminates the problem 
of left-censoring), the teacher’s CL award status, annual salary, and highest degree attained. 

Districts that choose to participate in the CL program are on average observationally 
different from districts that choose not to participate. Table 3 summarizes the characteristics 
of participating and non-participating districts in 1997-98, the midpoint of the analysis 
period. On average, districts that participate in the CL are smaller than non-participating 
districts, with average enrollment of 1,108 students, compared to 2,411 for non-participating 
districts. Participating districts are also less urban and more mral, with a smaller percentage 
of African-American students and higher percentage of white students, and have lower 
median household incomes than non-participating districts. 
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Table 3. Characteristics of Participating and Non-Participating Districts in 1997-98 



Characteristic 


Participating 

Districts 


Non-participating 

Districts 


Enrollment (number of students) 


1,108 


2,411 


Race/ethnicity of students 


Percent white 


96.7% 


91.1% 


Percent African-American 


2.0% 


7.6% 


Percent Hispanic 


0.7% 


0.7% 


Percent economically disadvantaged 


43.5% 


38.4% 


Teacher annual salary 


$27,939 


$28,511 


Teacher experience level (years) 


12.1 


12.4 


Student-teacher ratio 


13.1 


13.2 


Urbanicity 


Large or mid-size city 


8.0% 


19.9% 


Large or small town 


17.5% 


1 8.2% 


Rural area 


74.5% 


61 .9% 


County characteristics from Census data 


Percent urban 


1 5.5% 


31.8% 


Percent of adults with no college degree 


65.4% 


61 .5% 


Percent poor 


14.4% 


1 2.4% 


Median household income 


$31,945 


$36,039 


Propensity score 


0.665 


0.363 


Number of districts 


286 


236 



Because participating and non-participating districts are so different in their observable 
characteristics, and because those characteristics may be associated with differences in 
teacher mobility and retention that have nothing to do with the CL program, it is important 
to control as much as possible when examining teacher mobility. One measure of the 
average difference between participating and non-participating districts is the propensity 
score measure in Table 3, which measures the predicted probability of the district 
participating in the CL program in that year, based on the observable characteristics of the 
district. Participating districts have an average predicted participation probability of 66.5 
percent, more than 30 percentage points higher than non-participating districts. This means 
that we were able to explain and predict CL participation using the observable variables. 

Table 4 summarizes the average district and teacher characteristics of participating and 
non-participating districts, weighted by the number of unique full-time teachers observed at 
least once in that district during the analysis period. The teacher-weighted percentage of 
African-American students is lower in participating districts (9.1 percent) than non- 
participating (21.3 percent). Average teacher salary is also lower in participating districts 
($33,240) than non-participating districts ($40,907). Teachers in participating districts are in 
districts with higher state funding matching rates, lower median household income, and 
lower percent large or mid-sized city than teachers in non-participating districts. 
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Table 4. Description of Analysis Sample, Teacher Level 







Teachers in 


Teachers in Non- 






Participating 


participating 


Characteristic 


All Teachers 


Districts 


Districts 


District and Census Data 








Enrollment (number of students) 


9,470 


5,489 


12,124 


Percent African-American students 


16.4% 


9.1% 


21 .3% 


Percent Hispanic students 


1 .3% 


1 .6% 


1 .0% 


Percent economically 


38.0% 


40.9% 


36.1% 


disadvantaged students 
Teacher salary (2007 dollars) 


$37,840 


$33,240 


$40,907 


Percent urban 


63.7% 


45.6% 


75.7% 


State funding matching rate under 
pre-1996 rules 


58.0% 


67.5% 


51 .9% 


Percent of adults with no college 
degree 


53.2% 


58.7% 


49.5% 


Poverty rate 


12.3% 


1 3.9% 


1 1 .3% 


Median household income 


$39,025 


$34,162 


$42,267 


Percent in a large or mid-size city 


52.1% 


33.4% 


64.5% 


Percent in a large or small town 


20.4% 


23.8% 


18.2% 


Teacher Outcomes 








Average teaching spell duration 


5.1 years 


5.2 years 


5.0 years 


Promoted within district 


18.0% 


18.3% 


17.9% 


Moved to different MO district 


15.1% 


1 6.2% 


14.3% 


Left teaching or MO data 


38.0% 


34.8% 


40.2% 


Number of teachers 


146,308 


58,514 


87,794 



Table 4 also shows the outcome variables of interest and how they vary by CL status. 
The percentage of teachers whose teaching spell in the district ends with promotion within- 
district is slightly higher for teachers in participating districts (18.3 percent) than non- 
participating districts (17.9 percent), but teachers in participating districts moved to a 
different district in Missouri at a higher rate (16.2 versus 14.3 percent) and left teaching (or 
left the Missouri data) at a lower rate (34.8 versus 40.2 percent). The next chapter presents 
findings from different versions of the Cox regression model to determine whether they are 
likely the results of CL itself. 
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Chapter III 



Results 



H ere we present analysis results focusing separately on the questions of retention in 
the district and retention in the profession. As discussed in Chapter II, we consider 
those who change districts but remain in a teaching position in a Missouri public 
school to be “movers” and those who are no longer on the payroll of a Missouri public 
school to be “leavers.” This chapter presents findings from the preferred, benchmark model, 
the subgroup findings, and then discusses the sensitivity of the findings to different 
assumptions. 

A. Retention in the District 

We first analyzed the effects of CL participation on teacher retention in the district. 
From Table 4 we see that a larger percentage of teachers in CL districts are observed moving 
to a different Missouri district, with 16.2 percent of teachers in CL districts moving, 
compared to 14.3 percent of teachers in non-CL districts. This is not evidence of a causal 
relationship between CL and teacher retention in the district, as CL districts tend to be 
different from non-CL districts in a number of ways including wealth, size, and urbanicity, 
which could also explain differences in teacher retention. The regression model (discussed in 
Chapter II) controls for these effects, yielding an estimated impact of district CL 
participation holding other district characteristics equal. 

Controlling for other factors potentially related to teacher mobility (listed in Table 5), 
we found that teachers in CL districts were less likely to leave the district than those in non- 
CL districts. Table 5 shows the estimated effects on teacher retention in the district (the 
opposite of “exit by moving”) from our benchmark model, using a two-stage instrumental 
variable (IV) approach to control for district selection into CL participation. The estimated 
hazard ratio of 0.85 for district CL participation, which is statistically significant at the 0.01 
level, implies that teachers in participating districts are 0.85 times as likely as those in non- 
participating districts to move to a different district, all else equal. Table 5 also shows the 
estimated effects of the control variables in the model, with a lower district enrollment, 
higher percentage African-American students, lower percent Hispanic students, and lower 
district average teacher salary all significandy associated with a higher probability of teachers 
leaving the district. 




14 



Table 5. Results of Cox Proportional Hazard Model, Benchmark Estimates (Hazard 
Ratios) 



Explanatory Variable 


Exit by 
Moving 


Exit by 
Leaving 


Exit by Moving or 
Leaving 


District participates in CL 


.852** 


.899* 


.853** 


Log of district enrollment 


.814** 


.972* 


.921** 


Percent African-American 


1.005** 


.999 


1.001 


Percent Hispanic 


.981* 


1.041** 


1.026** 


Percent economically disadvantaged 


1.000 


1.002 


1.002* 


Log of district average teacher salary 
(2007 dollars) 


.588** 


.787** 


.666** 


Percent urban 


.999 


1.000 


1.000 


Percent with no college 


1.359 


.453** 


.592** 


Percent poor 


.871 


1.492 


1.208 


Log of median household income 


1.087 


.872 


.952 


Large or mid-size city 


.904 


.974 


.960 


Large or small town 


.926 


1.031 


.984 


Number of exits 


22,042 


55,645 


77,687 


Number of teachers 


146,308 


146,308 


146,308 



* Indicates significance at 5%; ** at 1%. 



Figure 1 illustrates the predicted survival functions for teachers in CL and non-CL 
districts. The height of the survival function is the predicted proportion of those still in 
teaching who remain in their original district, and the slope is the predicted probability of 
moving. Teachers are generally more mobile early in their career, which is reflected by the 
higher exit probability early in their tenure in the district. The higher survival and lower 
estimated exit rate for teachers in CL districts are reflected by the higher position of the 
survival curve and its more gradual slope for CL districts, compared to that for non-CL 
districts. For example, after ten years, the model predicts that 81 percent of teachers will 
remain in CL districts versus 77 percent in non-CL districts. After 20 years, the model 
predicts that 75 percent of teachers remain in CL districts versus 70 percent in non-CL 
districts. 

We also examined CL effects separately for different levels of teacher experience. 
The Cox model assumes that the CL effect is constant over the course of the teacher’s 
career. It would be reasonable to relax this assumption on the basis that eligibility for the 
incentives provided by the CL program depend in part on the teacher’s experience level. 
Eligibility for and potential size of the bonuses is tied to stages in the program, which itself is 
tied to minimum years of service (5, 7, and 10 years, respectively, for the three stages). For 
example, one might assume that the expected present value of future benefits would be a 
weaker incentive to reduce early career attrition than the bonuses being paid in current years 
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for teachers in their fifth to fifteenth year of teaching, when teacher participation is at its 
highest. 

Figure 1. Predicted Survival Until Moving, Using Benchmark Model 




Non- CL districts CL districts 



The first column of Table 6 shows the estimated effect of district CL participation for 
teachers in different experience ranges. The first row is the overall effect from Table 5. The 
second row is the estimated effect on the probability of teachers leaving within their first five 
years in the district. The estimated effect of 0.89 is not statistically significant at the 0.05 
level, indicating that there is insufficient data to conclude that district CL participation is 
associated with a lower probability of teachers moving during their first five years in the 
district. Similarly, the third row shows the estimated hazard ratio for leaving during years 6— 
10 in the district, estimated only from the sample of teachers that lasted at least 6 years in the 
district. The estimates of CL effects for more experienced teachers are necessarily less 
precise because the estimate for each experience range relies on a smaller and smaller 
sample. For example, nearly all teachers are observed in the first through fifth years of 
teaching, but a smaller number “survive” to more than 10 or more than 20 years of service. 

The only teacher experience range where the estimated hazard ratio that is statistically 
significant is for teachers with 1 1-20 years of experience. For these teachers the hazard ratio 
estimate is 0.56, implying that, for teachers with at least 11 years of experience in a CL 
district, the likelihood of their moving to a different district before they reach 21 years of 
experience is slightly greater than half what it would be in a non-CL district. This is the 
group of teachers that are eligible for the largest CL bonuses and who have the most to lose 
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by leaving the participating district. For teachers with at least 21 years of experience the 
estimated hazard ratio is also less than one, but not statistically significant, as we observe few 
teachers switching districts this late in their careers. 

Table 6. Results of Cox Proportional Hazard Model by Teacher Experience 



Outcome 



Risk period 


Exit by Moving 


Exit by Leaving 


Overall 


.852** 

22,042 exits 
146,308 teachers 


.899* 

55,645 exits 
146,308 teachers 


Exit within 1-5 years of starting 
teaching 


.888 

16,140 exits 
146,308 teachers 


.873** 

30,432 exits 
146,308 teachers 


Exit within 6-1 0 years of starting 
teaching 


.862 

3,296 exits 
71 ,680 teachers 


.786** 
5,965 exits 
71 ,680 teachers 


Exit within 11-20 years of 
starting teaching 


.564** 
2,139 exits 
48,751 teachers 


.912 

6,055 exits 
48,751 teachers 


Exit within 21 + years of starting 
teaching 


.708 

467 exits 
23,160 teachers 


1.126* 
13,193 exits 
23,160 teachers 



* Indicates significance at 5%; ** at 1% 

B. Retention in the Profession 

We repeated the analysis for retention of teachers in the profession as defined above. 
The raw differences in retention rates (Table 3) suggest that teachers in CL districts stay in 
teaching at higher rates (34.8 percent) than those in non-CL districts (40.2 percent). After 
controlling for district characteristics with the Cox model, the estimated hazard coefficient 
for the CL effect on leaving teaching is 0.9 and statistically significant at the 0.05 level (see 
Table 5, second column). This implies that teachers in CL districts are less likely (90 percent 
as likely) to leave teaching as those in non-CL districts, all else equal. 

This difference is illustrated graphically in Figure 2, showing the predicted survival 
functions for CL and non-CL districts. This figure shows how the likelihood of a teacher 
leaving the profession is highest for teachers in their first few years of teaching, and for 
teachers nearing retirement, with more than 20 years of experience. The model predicts that 
all else equal, 55 percent of teachers in CL districts will still be teaching there after 10 years, 
versus 51 percent of teachers in non-CL districts. For teachers with more than 20 years of 
experience the survival functions converge, reflecting a higher probability of leaving teaching 
for teachers in CL districts over this experience range (where all teachers approach 
retirement age). 
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Figure 2. 



Predicted Survival Until Leaving, Using Benchmark Model 
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Non- CL districts CL districts 



The second column of Table 6 shows separate CL effects on teacher retention in the 
profession for those at different experience levels. Unlike the effects on teacher retention in 
the district, here the largest and most significant effects are found in the lower experience 
ranges, with estimated hazard ratios of 0.87 for teachers in years 1-5 in the district, and 0.79 
for teachers in years 6-10. Years 6-10 are when teachers typically become eligible for CL 
awards, which combined with the potential for higher bonuses later in their career could 
explain why the retention effect in the profession is largest for this experience range. The 
estimated effect for teachers with 21 or more years of experience is actually 1.13, indicating 
that these teachers have a greater chance of leaving the profession if they are in a CL district. 
However, as we see in our sensitivity tests, this result is not robust to alternative 
specifications. 

C. Robustness of the findings 

The findings reported above were robust to several possible alternative assumptions and 
model specifications. 

For the analysis of retention in the district we treated leavers and promoted teachers as 
censored observations; for the analysis of retention in the profession we treated promoted 
teachers as censored observations. The findings were robust, however, to alternative 
specifications of the failure condition. For example, we repeated the analysis with mover or 
leaver as the failure condition and obtained a hazard coefficient estimate for CL districts that 
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lay between the two estimates for movers and leavers respectively (see Table 5, third 
column). 

Table 7 shows the results of some of the sensitivity tests, both overall and for different 
teacher experience categories. The first two columns estimate the model using a Weibull 
hazard function instead of a Cox hazard function. The Weibull distribution is typically used 
when hazard rates either increase or decrease exponentially with time. The results from this 
model are similar to those from the Cox proportional hazard model. 

Columns 3 and 4 of Table 7 use the standard Cox model, but omit teachers in districts 
that fall outside the region of common support, based on estimated propensity scores for 
district participation. This omits teachers in 38 non-CL districts that have propensity scores 
that are lower than those for any participating districts. These districts look quite different 
from others in the sample, being primarily wealthy urban districts with high property values 
per pupil. When these districts are omitted, the effect on retention in the profession for 
teachers with 21+ years of experience becomes insignificant, implying that that effect is 
being driven by those few unusual districts. Otherwise, the results look similar to those from 
the benchmark specification. 

The third set of sensitivity tests in Table 7 omits the two largest school districts, Kansas 
City and St. Louis, from the estimation sample. These two districts have their own pension 
systems, which could lead to different impacts on teacher retention. None of the results 
change significantly when these two districts are omitted. 

In addition to these checks, we also estimated the benchmark model using different 
time horizons, measured in years since the teacher began teaching in the district, for defining 
district CL participation. The benchmark model considers the district as a CL participant if it 
participated within five years of the teacher’s first year in the district (regardless of whether 
the teacher remained in the district during that time). We estimated the model using a 2-year 
horizon, a 10-year horizon, and the maximum observed value over the entire time period in 
our data. For each of these methods the results were very similar to the benchmark results 
shown in Table 5. 
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Table 7. Results of Cox Proportional Hazard Model, Alternate Specification 





Non-Cox Parametric Survival Curve 
Model (Weibull Distribution) 


Sample Restricted to Region of 
Common Support Using 
Propensity Score 


Kansas City and St. Louis School 
Districts Omitted 


Hazard Coefficient 


Exit by Moving 


Exit by Leaving 


Exit by Moving 


Exit by Leaving 


Exit by Moving 


Exit by Leaving 


Overall 


.811** 

22,042 exits 
146,308 tchrs 


.868** 

55,645 exits 
146,308 tchrs 


.825** 

17,467 exits 
103,352 tchrs 


.861** 

37,204 exits 
103,352 tchrs 


.846** 

20,613 exits 
130,745 tchrs 


.907* 

48,521 exits 
130,745 tchrs 


Exit within 1-5 years of 
starting teaching 


.831** 

16,140 exits 
146,308 tchrs 


.823** 

30,432 exits 
146,308 tchrs 


.875 

12,998 exits 
103,352 tchrs 


.817** 

21 ,913 exits 
103,352 tchrs 


.879 

15,276 exits 
130,745 tchrs 


.876* 

26,793 exits 
130,745 tchrs 


Exit within 6-1 0 years of 
starting teaching 


.807* 

3,296 exits 
71 ,680 tchrs 


.747** 
5,965 exits 
71 ,680 tchrs 


.768* 

2,536 exits 
47,933 tchrs 


.789** 
3,876 exits 
47,933 tchrs 


.846 

3,011 exits 
63,417 tchrs 


.793** 
5,063 exits 
63,41 7 tchrs 


Exit within 11-20 years 
of starting teaching 


.548** 
2,139 exits 
48,751 tchrs 


.904 

6,055 exits 
48,751 tchrs 


.548** 
1,623 exits 
31 ,640 tchrs 


.909 

3,981 exits 
31 ,640 tchrs 


.572** 

1 ,945 exits 
42,938 tchrs 


.919 

5,269 exits 
42,938 tchrs 


Exit within 21 + years of 
starting teaching 


.717 

467 exits 
23,160 tchrs 


1.126* 
13,193 exits 
23,160 tchrs 


.728 

310 exits 
14,048 tchrs 


1.042 

7,434 exits 
14,048 tchrs 


.769 

392 exits 
20,236 tchrs 


1.111* 

1 1 ,406 exits 
20,236 tchrs 



Indicates significance at 5%; ** at 1% 




Chapter IV 



Conclusion 



A. Summary and Policy Implications 

This paper presented evidence suggesting that a school district’s participation in the 
Missouri Career Ladder Program would tend to increase retention in the district and the 
profession, especially for mid-career teachers. 

The results reported here are especially useful when combined with other findings on 
this program. Previously reported analyses using 10 years of district-level test score data 
failed to find evidence for an impact on test scores (Booker and Glazerman 2009). However, 
focus groups and interviews conducted by Silman and colleagues (Silman and Glazerman 
2009) suggest that the program is popular with participating teachers and is run almost 
entirely by teachers themselves. Thus, when considering the benefits, there was no 
measurable test score effect, but a positive effect on retention and possibly a positive effect 
on teacher well-being generated by increasing their incomes by an amount that likely exceeds 
their hourly rate for the extra work completed. 

These findings have several implications for the teaching profession. First, they suggest 
that small bonuses can affect behavior. Some evidence suggests that incentive payments 
probably need to exceed 25 percent of teacher salary in order to neutralize the effects of 
turnover in hard to staff urban schools (Hanushek et al. 2004). If the Missouri CL program 
can reduce turnover with a bonus on the order of 10 to 20 percent of average teacher salary, 
even if by a modest amount, then it suggests that there may be more scope for policymakers 
to make a difference, even with limited budgets, than was previously believed. 

Second, the magnitudes of the effect estimates remind us that one should not expect 
large effects from small bonuses. The maximum payout of $5,000 has never been increased 
or adjusted for inflation. Consequently its purchasing power has eroded in value by almost 
60 percent. 

Third, the findings suggest that a bonus program that is not tied to student performance 
but offers additional pay for more work for mid-career teachers can make teachers feel 
better off even if it does not improve their teaching in an easily measurable way. 
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B. Limitations of the Study 

It is important to recognize the strengths and limitations of this research. The first 
caveat is that this study used econometric methods to address the selection bias that arises 
from unobserved differences between districts that did and did not choose to participate in 
CL. We believe this approach was well suited to this situation, but such methods do not 
provide the level of certainty that we would have from a well-run randomized controlled 
experiment. In an experimental setting, the researcher would randomly determine which 
districts would or would not implement CL, and the differences between the two groups 
could only be attributed to the program. In this case, there may still be unobserved 
determinants of CL status that could be confounded with our estimates of the program 
effects. 

Second, it is important to interpret the findings with a firm understanding of the 
program itself. The Missouri program was evaluated as an intact policy package. We assume 
the possibility to earn teacher bonuses was a chief motivating force, but the program also 
had elements of teacher empowerment, allowing teachers to mn the program themselves 
and to design their own work plan by which they could earn bonuses. There was an 
evaluation system in place, even if it may not have been implemented uniformly. Therefore, 
readers should use caution in applying these findings to other programs that use differendy 
structured incentives. To gain a more in-depth perspective on the Missouri program, see 
Silman and Glazerman (2009). 

A final thought to keep in mind when interpreting the results of our analysis is that we 
have no measure of what non-participating districts were doing instead of participating in 
CL. These districts could be using those funds to supplement teacher salary generally, but it 
was not possible to measure all the possible teacher retention strategies in place in CL and 
non-CL districts. Thus we estimate an overall effect of CL participation relative to all non- 
participating districts, regardless of the programs those districts may have implemented. 

In the future, Missouri will have more detailed test score data, including scores from 
consecutive years and consecutive grades, which was not available for this study. It may also 
be possible to measure more of the factors that determine participation and that describe 
what districts do in the absence of CL. Combining this newly emerging data with more 
detailed information on the spatial distribution of school districts presents a promising 
avenue for more policy-relevant research that can build on the current findings and thus 
drive good decision-making. 
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